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^performance,  aid  with  software  debug,  and  assist  with  predicting  the  impact  of 
proposed  system  modifications.  The  system  produces  automated  reports  and 
graphs,  several  types  of  which  are  appropriate  for  viewing  by  management. 

This  document  is  intended  to  conmunicate  to  the  TRIDENT  community  the 
overall  capabilities  of  the  system,  and  its  potential  worth  in  life  cycle 
support  of  the  present  system  and  constructing  models  for  future  systems. 
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FOREWORD 


The  Verification  and  Evaluation  System  for  TRIDENT  (VEST):  Summary  to  Man¬ 
agement  is  intended  for  program  managers  and  lead  engineers.  It  summarizes  those 
aspects  of  VEST  which  are  believed  to  be  of  interest  to  those  directing  and 
planning  TRIDENT  Fire  Control  Projects. 

In  terms  of  content,  this  summary  is  of  a  higher  level  than  the  VEST  Con¬ 
cepts  and  Capabilities  Document  and  the  VEST  Development  Specification,  which  may 
be  referred  to  for  more  specific  technical  details  about  the  capabilities  of  the 
system. 

For  further  information  on  VEST  or  to  direct  comments  concerning  this  pub¬ 
lication,  contact  the  Naval  Surface  Weapons  Center  (NSWC),  FBM  Geoballistics 
Division,  Code  K54,  Dahlgren,  Virginia  22448. 
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BACKGROUND 

Throughout  the  life  cycle  of  real-time,  embedded  computer  systems,  numerous 
situations  arise  which  require  access  to  detailed  system  performance  data.  The 
evolutionary  nature  of  embedded  computer  systems  often  requires  them  to  be  modi¬ 
fied  as  well  as  maintained.  Given  this,  coupled  with  the  additional  problem  of 
verifying  and  validating  delivered  software  and  changes  thereto,  and  the  need 
often  to  recreate  and  *olve  obscure  problems  reported  from  the  field,  there  is 
considerable  justification  for  a  system  like  the  Verification  and  Evaluation 
System  for  TRIDENT  (VEST) . 

In  1973  when  design  of  the  TRIDENT  Fire  Control  System  was  begun,  it  was 
clear  that  the  inaccessibility  of  certain  performance  data  had  compounded,  if  not 
directly  caused,  many  problems  with  earlier  systems.  Real-time  software  appeared 
to  be  a  recurring  problem  area  in  which  there  rarely  seemed  to  be  a  comfortable 
amount  of  performance  data.  Such  software  difficulties  were  most  always  related 
to  hardware  operating  characteristics.  Often  the  interrelationships  could  be 
discovered  only  after  problem  identification  through  painstaking  investigation, 
sometimes  requiring  design  of  unique  measurement  configurations.  To  attack  these 
problems,  the  decision  was  made  to  design  a  passive  Performance  Evaluation  Inter¬ 
face  (PEI)  into  the  TRIDENT  Digital  Control  Computer  (DCC).  This  interface  was 
to  provide  data  quantities  including  Program  Status  Word  (PSW) ,  instruction  being 
executed,  memory  location  being  referenced  and  contents  thereof,  and  results  of 
instruction  execution,  to  an  external  port  at  minor  cycle  clock  speeds.  This 
interface  was  later  used  by  the  VEST  measurement  system. 


GENERAL  APPROACH  TO  THE  PROBLEM 

Software  is  usually  delivered  after  being  tested  by  a  relatively  small  set  of 
test  cases,  and  this  leaves  a  host  of  unknowns. 

First,  there  is  the  question  of  test  coverage.  The  criteria  for  adequacy  of 
software  testing  is  not  established,  but  management  could  gain  some  confidence  if 
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it  knew  that  at  least  a  minimal  test  set  had  been  used,  say,  to  cover  all 
branches  of  the  program.  Such  branch  tracing  is  one  of  the  possibilities  with 
VEST. 

How  well  does  the  system  perform?  The  answer  to  this  question  is  usually 
not  immediately  apparent  because  the  true  worst  case  has  probably  not  been  well 
defined  for  every  time  frame.  The  difficulty  in  setting  up  realistic  loads  and 
being  able  to  closely  monitor  some  of  the  subtle  interactions  of  the  hardware  and 
software  makes  it  essential  that  better  measurements  be  taken  than  have  usually 
been  done  in  the  past.  One  typical  performance  measurement  is  the  closeness 
to  time  margins,  i.e.,  the  performance  in  meeting  response  time  requirements  and 
deadlines.  These  and  other  quantitative  measurements  such  as  idle  time  and 
memory  activity  can  lead  to  an  answer  to  the  question  of  reserve  capacity. 

The  lack  of  detailed  knowledge  about  the  actual  inner  workings  of  the  soft¬ 
ware  tends  to  hide  trouble  areas  and,  first  of  all,  make  it  difficult  to  identify 
candidates  for  improvement.  Then,  if  any  modifications  are  proposed,  there  is 
the  problem  of  determining  the  impact  of  modification. 

The  above  uncertainties  result  in  a  Mleave  it  alone"  syndrome,  and/or  modi¬ 
fications  by  "trial  and  error,"  and,  of  course,  the  receipt  of  many  Field  Trouble 
Reports  (FTRs)  soon  after  the  software  reaches  the  operating  forces.  A  better 
approach  is,  after  the  system  is  developed,  to  measure  and  analyze  it,  and  docu¬ 
ment  the  results.  This  should  verify  test  coverage,  determine  closeness  to  time 
margins,  identify  anomalies  and  bottlenecks,  quantify  reserve  capacity,  and 
identify  improvements.  For  future  systems  or  modifications  to  the  current,  the 
above  results  should  act  as  a  guide  and  also  be  used  to  construct  models  to 
assist  future  design  efforts.  (See  Figure  1.) 

THE  VEST  SOLUTION 

This  system  permits  a  complete  analysis  of  the  system  behavior  under  real¬ 
time  operational  scenarios.  This  is  expected  to  result  in  the  following  benefits: 


MEASURE  & 
ANALYZE 


Figure  1.  The  VEST  Approach 


1.  Enhanced  software  verification  and  validation  (V&V)  through  verified 
program  coverage,  path  coverage,  correct  program  sequencing,  and  coordinated 
software  deadlines. 

2.  Identification  of  reserved  capacity  and  growth  potential  of  the  system, 
and  conversly  potential  bottlenecks,  by  characterization  of  utilization  levels, 
closeness  to  time  margins,  and  system  device  idle  time. 

3.  Reduced  software  costs  for  investigating  sporadic  malfunctions  by  pro¬ 
viding  a  trace  of  hardware  and  software  activity  immediately  preceding  the  fail¬ 
ure  which  will  lead  to  quicker  problem  resolutions. 

4.  Quantification  of  system  performance  parameters  needed  to  drive  models 
which  in  turn  can  provide  low-cost,  quick- response  solutions  to  performance 
trade-offs  involving  hardware  and  software  modifications. 

5.  Support  of  development  of  future  versions  of  the  fire  control  system 
(FCS)  by  characterizing  workloads  and  utilization  levels  so  that  future  designs 
can  concentrate  on  resources  necessary  to  optimize  performance-pacing  tasks. 

DESCRIPTION  OF  THE  BASIC  SYSTEM 

VEST  consists  of  two  main  portions:  (1)  an  online  portion  which  instru¬ 
ments,  collects,  and  records  performance  data  about  TRIDENT  Fire  Control  System 
activity  and  (2)  an  offline  portion  which  reduces  and  analyzes  this  data  and 
generates  reports.  The  online  portion  of  VEST  includes  special  purpose  hardware 
for  acquiring  and  processing  signals  which  reflect  system  activity  and  the  Data 
Collection  Software  (DCS)  for  accumulating  and  recording  such  results  on  magnetic 
tape.  The  hardware  (Figure  2)  includes  a  specially  made  Event  Trace  Unit  (ETU) 
and  an  off-the-shelf  Commercial  Monitor  (CM) .  The  offline  portion  of  VEST  is 
the  Data  Reduction  Software  (DRS)  and  associated  utility  programs  which  run  on 
NSWC's  CDC  6700  computer  system  and  process  tapes  produced  by  the  online  portion. 


PROBES 


Figure  2,  The  VEST  System 

The  ETU  shown  in  the  center  of  Figure  2  is  a  microprocessor-controlled  de¬ 
vice  ,  the  detailed  design  and  implementation  of  which  was  done  by  NSWC  from  a 
preliminary  Hughes  Aircraft  Company  design.  It  interfaces  with  the  Performance 
Evaluation  Interface  (PEI)  of  the  DCC  and  can  capture  data  at  CPU  clock  speed. 
Probe  data  of  discrete  electrical  events  from  other  parts  of  the  computer  group 
and  FCS  are  available  to  it  and  also  to  the  CM. 

The  performance  data  collected  by  the  ETU  is  sent  to  the  SDS’s  PDP-11  mini¬ 
computer  for  recording  on  tape.  As  indicated  in  Figure  2,  selectable  data  from 
the  ETU  may  be  passed  to  the  CM  and  vice  versa. 

The  DRS  reduces  the  measurement  data  collected  by  the  Software  Development 
System  (SDS)  in  order  to  generate  meaningful  reports  which  characterize  FCS 
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activity.  CM  tapes  are  also  processed  on  the  central  computer  by  its  own  DYNAPAR 
proprietary  package. 

The  ETU  has  within  it  an  Associative  Memory  (AM)  which  allows  it  to  "match" 
any  of  its  contents  with  the  high-speed  data  stream  from  the  CPU  and  CM.  These 
data  streams  may  represent  locations  of  code  or  data  in  the  CPU  or  any  pattern 
supplied  by  the  CM.  An  AM  allows  simultaneous  comparisons  within  the  whole 
memory  bank  to  be  carried  out  on  every  desired  bit  of  the  word  being  compared. 
The  equipment  signals  which  memory  locations,  if  any,  had  a  "hit."  These  AM 
devices  are  often  referred  to  as  "content  addressable"  memory  (i.e.,  tell  the 
address  in  which  the  content  is  located).  The  initialization  of  each  VEST  ex¬ 
periment  establishes  the  significance  of  each  AM  address. 

The  ETU  can  also  detect  preselected  external  events  sensed  by  probes.  The 
events  are  subsequently  recognized  by  the  DRS  software  as  significant  milestones 
in  the  progression  through  the  operational  sequence.  These  come  either  directly 
from  fire  control  or  via  the  CM  programmable  logic  patchboard.  Patchboard  fea¬ 
tures  on  the  CM  include  logic  elements  such  as  AND,  OR,  NOT,  latches,  flip-flops, 
fanout  units,  decoders  and  the  like.  The  CM  is  in  itself  a  powerful  measurement 
device  which  is  preferred  for  certain  instruction  mix,  CPU  throughput  and  other 
studies . 

An  important  capability  of  the  ETU  is  tracing  software  activity  to  an  extent 
which  was  not  possible  with  previous  equipment.  It  allows  tracking  software 
tasks,  modules,  and  CPU  state  changes.  Monitoring  the  activity  of  certain  Launch 
mode  programs  or  differentiating  between  the  Monitor  and  the  Executive,  or  just 
simply  limiting  the  data  collected  to  one  portion  of  the  software,  are  just  a  few 
of  the  things  now  possible.  VEST  can  also  trace  program  branches  and  procedure 
invocations  and  thereby  verify  program  coverage  and  correct  sequencing.  This 
also  facilitates  problem  area  investigation. 

The  CM  is  important  for  PEI  as  well  as  non-PEI  signals  and  provides  patch¬ 
board  logic  and  additional  data  recording  modes.  It  is  needed  for  accumulating 
high-speed  mappings  (distributions)  for  such  things  as  instruction  mix  and  CPU 


throughput  studies.  Also,  it  can  be  used  in  conjunction  with  the  ETU  for  Input/ 
Output  (I/O)  contention  and  I/O  utilization  measurements.  In  instruction  mix 
runs,  it  would  use  PEI  data  passed  to  it  by  the  ETU.  As  mentioned  earlier,  a 
major  feature  of  the  CM  is  its  ability  to  logically  combine  signals  via  the 
various  logic  elements  contained  on  its  patchboard. 

From  the  user  terminal,  the  ETU  is  initialized  to  collect  the  desired  param¬ 
eters  from  the  FCS.  The  specification  for  collection  can  be  quite  sophisticated, 
including  calling  out  raw  PEI  data,  software  category  change  events  (that  is, 
changes  in  CPU  state,  task,  and  module),  execution  of,  or  reference  to,  specific 
program  locations,  procedure  trace,  branch  trace,  and  the  like.  Initial  condi¬ 
tions  or  "set-ups"  can  be  stored,  or  "canned,"  and  reused  whenever  desired.  With 
the  operators  ability  to  override  or  add  to  certain  portions  of  the  canned  pro¬ 
cedures,  the  system  is  convenient  yet  flexible.  Selected  printouts  of  collected 
data  is  also  an  option. 


ECONOMY  IN  DATA  REDUCTION 


This  automated  system  will  greatly  reduce  the  expense  of  engineering  and 
data-technician  work  on  reducing  the  data  from  the  measurement  runs.  In  addi¬ 
tion,  the  DRS  of  the  system  performs  an  "analysis"  of  the  reduced  data  and  pre¬ 
sents  it  in  graphical  and  tabular  form  for  easy  viewing  and  comprehension.  This 
is  covered  in  more  detail  in  the  next  section.  Further  economies  are  realized 
"up  front"  in  the  data  acquisition  phase.  The  measurement  hardware,  which  is 
under  program  control,  can  be  made  to  collect  only  certain  qualified  events  and 
thus  greatly  reduce  the  amount  of  data  that  the  remainder  of  the  system  has  to 
handle.  Furthermore,  there  are  provisions  to  permit  the  user  to  predefine  mul¬ 
tiple  starts  and  stops  of  data  collection  based  on  specific  portions  of  the  FC 
operational  sequence. 

The  DRS  has  the  capability  of  processing  the  collected  data  using  processing 
variables  (i.e.,  counters,  timers,  mappers)  and  producing  various  statistics  on 
them,  such  as  the  maximum,  minimum,  mean  and  standard  deviation.  These  results 
can  then  be  produced  in  standardized  or  user-defined  formats. 
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CLEAR  AND  EASILY  UNDERSTOOD  REPORTS 


This  system  supplies  reports  which  are  clear  and  easily  understood.  The 
output,  in  most  cases,  does  not  require  additional  engineering  hours  in  order  to 
gain  useful  information  from  the  output  statistics.  VEST  provides  for  selection 
of  a  large  number  of  different  graphical  and  tabular  presentations.  One  example 
(see  Figure  3)  is  a  Pie  Chart  rendition  of  software  utilization  results. 


Figure  3.  Software  Utilization  Report  (Pie  Chart  Format) 

There  are  also  the  Time  Line  Plots,  which  are  particularly  useful  in  depict¬ 
ing  FCS  activity  (see  Figure  4),  and  the  familiar  Gantt  Chart  (bar  graph)  form 
which  is  particularly  good  for  showing  periods  of  activity  of  different  hardware 
and  software  components.  There  are  other  formats,  such  as  the  histogram  (see 
Figure  5)  for  showing  distribution  of  activity,  Kiviatgraphs ,  Event  Listing, 
Statistical  Summary,  Exception,  and  Profile  reports.  The  user  may,  through  these 
options  and  report  generator  facilities,  build  any  report  format  desired. 
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Figure  4.  System  Line  Report  (Time  Plot  Format) 
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Figure  5.  Software  Utilization  Report  (Histogram  Format) 
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REDUCING  SET-UP  TIME  AND  ENCOURAGING  USE 


Verification  and  Validation  (V&V)  and  Perforaance  Evaluation  are  both  activ¬ 
ities  high  in  engineering  labor  cost  for  set-up  and  test  design.  Not  too  auch 
can  be  done  about  the  design  tiae,  but  auch  of  the  set-up  effort,  with  such 
tiae-consuaing  activity  as  identification  of  absolute  code  addresses  and  other 
preparatory  operations  typical  of  older  systeas,  has  been  circuavented  in  VEST. 
A  convenient  user  interface  with  a  specially  designed  high-level  language  is 
provided,  and  the  ability  to  select  syabolic  addresses  and  events  greatly  facili¬ 
tates  set-up.  In  addition,  libraries  of  "canned,"  often  used  experimental  set¬ 
ups  and  report  definitions  can  be  called  up  to  autoaatically  initialize  the 
systea  in  the  data  collection  and  data  reduction  phases.  These  features  should 
not  only  save  aoney,  but,  by  aaking  the  task  easier,  thereby  encourage  a  higher 
level  of  usage  of  the  VEST  systea. 


CONCLUSION 

VEST  represents  a  structured  approach  tc.  obtain  aore  accurate  and  revealing 
data  froa  eabedded  computer  systeas.  This  is  not  only  to  establish  the  perfora¬ 
ance  of  existing  systeas,  but  to  gain  aore  insight  into  the  problea  and  assist  in 
new  designs.  The  equipment  currently  in  place  represents  new  advances  in  the 
ability  to  aonitor  and  analyze  computer  systea  and  weapon  systea  activity. 
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